A storage system, either direct-attached or storage area networked (“SAN”) based, provides a block interface to a host. The block interface uses volume abstraction, where each volume represents a linear storage space in sectors. As technology advances, the size of each volume grows. The modern storage system volume is easily in multiple terabytes (“TB”).
Generally, a file system or application breaks the linear storage space into blocks and arranges the storage space. To improve the performance, the application writes multiple concurrent files at the same time, where each file can be placed at distinct locations on the storage space.
For instance, in a video surveillance application that supports many video camera streams, each camera stream can be written to one or more files. With additional cameras, multiple files can be written at the same time. This can be an example of a multi-stream application.
Another example of a multi-user application is a cloud storage application that uses the Internet to provide for on-line storage of user data. Many users can store their data or files at the same time, thus simultaneously writing files to a storage system, where all incoming data or files can be placed in distinct locations in the storage system.
The storage performance for multi-stream and/or multi-user applications is problematic due to performance limits of electromagnetic hard disk drives (“HDDs”) of the storage system. Although modern electromagnetic drives have outstanding sequential speed, over 100 MBps per drive and can linearly scale with drives, these electromagnetic drives suffer from random read/write performance limits, including slow seek time, rotational latency, speed variations between outer tracks and inner tracks, and other limits.
With respect to seek times, for SATA drives, the seek time is typically 10 ms; even with enterprise SAS drives, the seek time is around 5 ms. Additionally, for every disk seek, there is a disk rotational latency until the target sector reaches the disk head. For SATA drives with 7200 RPM, the average rotational latency is 4.16 ms.
Furthermore, storage quality of service (“QoS”) schemes can incur additional performance cost when dealing with the multi-stream and/or multi-user applications. For instance, RAID5 can incur a huge performance cost since a single write generates two additional reads and one additional write. This can happen with random traffic since RAID5 requires reading data and the parity for the data before writing the new data and the new parity. Consequently, read/write operations of data blocks are especially slow for multi-stream and/or multi-user applications and in particular for RAID.
This problem can be addressed by using many disks for parallel access or using high performance drives, such as SAS/FC drives and solid states drives (“SSD”). However, these are expensive solutions, and thus cost prohibitive.
Therefore, it is desirable to provide methods and systems for efficiently storing data blocks that allow for the use of existing low-cost, high-capacity SATA drives without the need to use many high-end drives (e.g., SAS/FC drives, SSD, and so forth).
Furthermore, it is desirable to provide methods and systems for reducing the performance variation of random read/write operations in storage systems due to mechanical latency in electromagnetic disk drives or RAID technologies.
Additionally, many multi-stream and multi-user applications send data blocks consistently to the storage unit. However, due to file system allocation and lack of communication with the storage unit, the data blocks from each stream may be written to the storage unit (otherwise referred to as “flushing” data to the storage unit), leading to overall reduced performance in the system. For instance, in video surveillance applications, each video stream can have different frame rates and/or video resolutions. Some streams are initiated by motion activated detection, while some streams are not. Without the knowledge of how fast the data is coming and when, the cache may not be flushed when the data block is full. Thus, instead of a few large blocks written to the storage stack, many small blocks are also sent down. Thus, as a result, the data blocks written to the storage unit are written to random locations on the storage unit, which requires the storage system to use more resources to process the data blocks.
Furthermore, for applications such as network surveillance recording (“NVR”) applications, which involve continuous writes, contiguous dirty buffers can accumulate over time. The accumulating area of memory can be called a “growing spot” since data is continually written to the memory. Depending on the bandwidth for each data stream from the NVR application, the accumulating speed for each growing spot can be different, which can require frequent writes from cache to the storage unit. Frequent flushing of growing spots can severely degrade overall performance. The degradation can be compounded even more when a number of growing spots occur, which is typically true for hundreds of NVR streams.
Generally with NVR applications, meta-data is created along with video data. Typically, meta-data has relatively smaller block size compared to video data and is randomly written to a virtual drive; whereas video data is much larger in block size and is written sequentially to the virtual drive. In order to update the meta-data, frequent writes to the storage unit are generated in a small area of the virtual drive, which causes cache hits (i.e., hitting the same area multiple times). These areas can be called “hot spots”. Frequent flushing of the hot spots can also severely degrade performance since write commands are required, which incur multiple seek times due to the randomness in writing the meta-data to the storage unit.
Current caching systems do not treat meta-data differently from video data, thus incurring enormous performance cost. Therefore it is desirable to provide methods and systems for storing data according to the type of data to be written to storage.